Status: draft

A document record is the archival unit of a Diaryx workspace: everything the workspace knows about one gathered document. It is the target shape that every importer emits, the structure that annotation lives in, and the referent that identifiers point at.

The design follows the PHILOSOPHY: gather, transcribe, annotate, publish — the same structure the Joseph Smith Papers Project uses for each document (artifact, transcript layers, source note, editorial annotation). Everything is plain text — Markdown with frontmatter for prose, YAML for the card that stands in for a file that is not prose — and a document record is fully legible in any text editor.

Components

A document record consists of:

1. Artifact — the original file, byte-for-byte (a scan, photo, PDF, audio file, or born-digital original). Never modified.
2. Attachment sidecar (the catalog card) — prov's <payload>.<ext>.yaml, minted beside the artifact by attach. It carries the artifact's title, its content: pointer, typed links, and provenance metadata. This is the record: there is exactly one document per artifact, and it is the sidecar.
3. Transcription(s) — zero or more Markdown files containing the artifact's textual content. Each is a document in its own right: independently editable, publishable, audience-filterable, and addressable.
4. Annotation — footnotes within a transcription, plus sibling note files for document-level commentary, linked from the catalog card.

Only the artifact and its sidecar are required. A born-digital Markdown import needs no transcription; a photo of a letter usually wants one.

One document per artifact

An earlier draft of this spec gave the artifact a Markdown catalog card of its own. That card was a second document about the same file, since attach had already minted the sidecar — two nodes in the tree, two rows in a lens, and no answer to which of them the record is. A transcription link would have had two places to hang off.

So the card is the sidecar, and there is no Markdown card. The cost is real and is accepted: a whole-file YAML document has no prose body, so the free-form source note is a frontmatter string (source_note:) rather than paragraphs under a fence. Anything longer than a note belongs in a sibling commentary file (see Annotation), which is where document-level prose was always meant to live.

Storage layout

An artifact and its sidecar are siblings in an attachments/ directory beside the index that contains them — the placement import_attachment writes and prov's reverse index requires (photo.jpg ⇄ photo.jpg.yaml is found by name alone, so the pair cannot be split). The containing index is resolved by the same rules a story written that day would use, so an artifact is filed under the period index the vault already files writing into, rather than into a folder of its own:

Daily/2026/
  index.md                              (the containing index; contents: links the sidecar by id)
  attachments/
    letter-from-may-1943.eml            (artifact, byte-for-byte)
    letter-from-may-1943.eml.yaml       (catalog card: content:, attachment: true, transcription:, provenance)
    letter-from-may-1943.transcript.md  (transcription)

There is no record folder and no date prefix. A record is a link neighbourhood, not a directory: the sidecar names its payload through content:, its transcriptions through transcription:, and its container through the index's spanning field. Tools must follow those links rather than assume any path — including attachments/, which is where writes land, not a location anything may rely on when reading.

Records are not confined to Documents/. That directory was this spec's earlier convention; placement now follows the vault's declared date view, and a user may move a record anywhere.

Born-digital text imports (a Day One journal, a folder of Markdown) have no artifact, and so no card and no attachments/ directory: they import as plain entries, filed into the same date structure by the same placement rules. A record is what an artifact gets; an entry is what text that was never anything else gets.

The attachment sidecar (catalog card)

Extends prov's sidecar — which attach mints carrying title, content: and attachment: true — with transcription links and provenance metadata. The whole file is the metadata; there are no fences:

title: Letter from Ruth Harris, May 1943
content: letter-1943.jpg
attachment: true
created: 2026-08-16
transcription:
- '[Letter from Ruth Harris, May 1943 (transcript)](letter-1943.transcript.md)'
source: Grandma's shoebox, box 2
source_note: >-
  Water-stained but legible. Ruth describes the week Dad was born.
  This is the only surviving letter from this period.
date_of_document: 1943-05-12
people:
- Ruth Harris
places:
- Ogden, Utah

Two keys are prov's and are not this spec's to redefine: content: is the pointer to the payload, resolved relative to the sidecar's own folder, and attachment: true is what stops prov reading the payload as prose. Containment is not a key here at all — the index gains the spanning link, in id-form, so the relationship survives the sidecar being moved.

The two dates answer different questions and must not be conflated. created is when Diaryx/prov made the document — an act of the software, stamped like any other document's. date_of_document is when the original was made, which is a fact about the world that often only its owner knows. A scan made today of a letter from 1943 has created: 2026-08-16 and date_of_document: 1943-05-12.

This matters most to importers, which are the tempting place to blur it: an EXIF origin date, an email Date: header, and a Day One entry's timestamp are all statements about the original, so they belong in date_of_document. created is never inferred from the source — it is when the record entered this archive, and nothing else.

Frontmatter keys

All provenance keys are optional. Unknown keys remain permitted, as everywhere in Diaryx frontmatter.

date_of_document: unknown

A date lens groups by a chain — [date_of_document, created, updated], declared per-vault as a view's group: — and the chain is first key present wins, not first key parseable. A grain then cuts the winning value, and validates rather than slices: banana at year grain is not the group bana, it is no group at all.

Those two rules compose into the marker. Writing

date_of_document: unknown

stops the chain at the first link and yields a value no grain can cut, so the document files as undated instead of inheriting the day it was scanned. Verified against prov: a document carrying date_of_document: unknown alongside created: 2026-01-01 lands in (ungrouped), while the same document without the marker lands in 2026.

This is what a shoebox needs. A folder of 200 undated scans imported on one afternoon would otherwise all inherit that afternoon through created, burying a year of the reader's real archive under a day that means nothing about any of the documents. Undated is the honest answer, and the marker is how a record gives it.

Two consequences worth stating plainly:

• The marker is a convention, not a mechanism. Nothing privileges the literal string unknown — any uncuttable value stops the chain identically, so a typo'd May 1943 silently files as undated too. unknown earns its place by saying deliberately unknown to the next human, not by doing anything special to the engine. (Nor does anything catch the typo: prov's check does not validate type: date values, which is a real gap and not this spec's to close.)
• date_of_document is already load-bearing for placement. Period indexes are resolved by the date they carry rather than by title — real ones are titled 2025-03 index, December 2025, March 2026 and 07, and no title rule matches all four — so the index-creating path stamps date_of_document on each index it makes. The key is structural, not merely descriptive, and a tool writing it is participating in filing.

Transcription files

A transcription is an ordinary Markdown file — fenced frontmatter and a prose body, unlike the card — with a back-link to that card:

---
title: Letter from Ruth Harris, May 1943 (transcript)
transcription_of: '[Letter from Ruth Harris, May 1943](letter-1943.jpg.yaml)'
transcription_status: needs_review
audience:
- family
---
Dear Mother,

We are all well here, though the rain has not let up since Tuesday...[^1]

[^1]: Spring 1943 was unusually wet in northern Utah.

Keys

Status lifecycle

Machine-produced text (OCR, speech-to-text) is a draft pending human review, never silently treated as truth:

• draft — machine-generated or in-progress, not yet human-reviewed
• needs_review — complete but awaiting verification against the artifact
• verified — a human has checked it against the artifact

This gives users the archivist workflow: gather → transcribe → verify → annotate → publish. verified is a position in that workflow, not an attributed claim about particular bytes — see Provenance below for what it does not yet say, and what prov will eventually let it say.

Multiple transcription layers

transcription is a list. A record may carry multiple layers — e.g. a verbatim transcript (original spelling, strikethroughs preserved) and a normalized reading copy — as separate files. No layer-type key is defined yet; distinguish layers by title until real usage motivates one.

Annotation

Two forms, both plain Markdown:

1. In-text annotation: standard Markdown footnotes inside the transcription file.
2. Document-level commentary: sibling Markdown files linked from the catalog card (via links or contents), for essays, research notes, or historical introductions about the document.

Explicitly out of scope: standoff annotation. No character-offset or range anchors into transcripts — offsets break on every edit. If in-text anchoring beyond footnotes is ever needed, anchor to quoted text snippets (fuzzy-matched), not positions.

Provenance

This spec's source key and transcription_status lifecycle predate prov's provenance proposal, which reaches for the same goals with more mechanism behind it. That proposal is an early draft — eight open questions, three of which change the shape of the rest, and a Phase 0 that is carry only — so nothing here changes today. What this section fixes is the layering, so that neither side grows a duplicate of the other.

transcription_status is not prov's verified, and should not become it

prov proposes verified: [{by, at, of}] — an append-only list of dated, attributed confirmations, each bound to the fixity digest it was made against — with trust tiers derived from the entries rather than stored. Diaryx's transcription_status looks like a smaller version of that, but the two answer different questions and both are worth keeping:

• transcription_status is a position in the transcription workflow. needs_review has no prov equivalent and will not get one: prov's proposal lists workflow as a non-goal ("no review queues, no assignment, no approval states").
• verified is an attributed, dated, digest-bound claim. It says who checked, when, and against which bytes.

The intended relationship is that the second backs the first, rather than replacing it: transcription_status: verified is legitimate only when a verified entry exists whose of matches the transcription's current digest.

The gap this leaves open today

Diaryx's verified is a bare enum, so it survives an edit to the transcription it describes. Someone checks a transcript against the artifact, someone else fixes a line, and the frontmatter still claims the transcript is verified — prov's proposal names this exact failure and calls it worse than having no vocabulary at all, "because it launders staleness as assurance."

Diaryx stamps fixity over attachments by default, and a vault can widen it to fixity: all in config.yaml, which is what puts a content_hash on a transcript's own body — so the ingredient is available, but a vault that wants checkable transcription claims has to ask for it. When prov's Phase 2 lands, the demotion rule comes with it: a transcript whose bytes no longer match the digest its verification was made against falls back to needs_review, and prov reports a VerificationStale finding. Until then this is a known, unenforced gap, not a solved problem — and transcription_status: verified should be read as "a human said so at some point," not as a checkable claim.

Two rules from the proposal apply to Diaryx as written, and are worth stating here because importers are the tempting place to break them:

• Nothing auto-promotes a transcription to verified. A clean OCR run, a confident speech-to-text score, and a passing prov check are all attestations about state; verification is a claim about meaning, and only a person or a named process makes one.
• A stored verification never suppresses a finding. You cannot verify your way out of a fixity mismatch.

source and prov's sources

source stays a free-form human string — Grandma's shoebox, box 2 is not a path and never will be. It is the narrow case of prov's proposed sources list.

The part worth tracking is §6 of the proposal: extending prov's fields mechanism from term-valued to path-valued fields, so that a frontmatter string prov knows is a path gets rewrite-on-move, locator handling, and dangling-reference reporting for free. It is Phase 1, independently valuable, and fixes a live bug rather than adding a capability. When a document record's provenance needs to name another document in the workspace — a finding aid, an accession record, a parent collection — that belongs in a path-valued sources field, not in source, and not in a new Diaryx-specific key.

Identifiers

Identity is prov's, not Diaryx's. This spec earlier described a hand-rolled ID index (Meta/Ids.md, reached through a Config indirection); prov settled the question first and differently, and that design was never built. What follows is the layer Diaryx actually runs on.

prov treats identity as a strictly-additive layer over a workspace that already works on paths alone: the graph, traversal, and mutation layers never dereference an ID, and turning identity off changes nothing about how a record is stored or linked. IDs are minted at registration rather than derived — an unregistered document has no opaque ID at all and is addressed by path. Two config axes govern it, both set in the vault's config.yaml:

• identity: none | lazy | eager — when an ID is minted. lazy (prov's default) registers on link-by-id or publish, keeping the registry to the minimal set something external depends on. Diaryx runs eager, which also registers on create: stable identity from birth, at the cost of a registry entry per document.
• id_storage: registry | frontmatter | both — where a registered ID lives. Diaryx runs both: each document carries its own id field and the registry document is a rebuildable cache. Frontmatter storage is what makes identity move- and copy-robust — the ID travels with the file, which is exactly what an artifact and its card copied out of the workspace need.

The registry is a first-class member of the workspace, discovered through prov's registry pointer relation off the root. It is not a Meta/ convention, and nothing in this spec should assume its path.

What remains Diaryx's is what publication means. Publication turns a registered ID into a permanent promise: at publish time the file receives a permalink frontmatter key holding the full URL with the ARK inside, in the canonical bare-ark form served by the site proxy (e.g. permalink: https://diaryx.org/ark/dxg6h4ncm/4k2ph9t, blades illustrative; the ark:99999 prefix form is accepted as an alias). The ARK's workspace blade is the vault's prov workspace_id, and its file blade is the document's prov id — prov and diaryx_ark mint the same 6-random-plus-check shape over the same alphabet, so publication mints nothing per file and stores no second identifier. The ID is thereby recorded redundantly — in the document, in the registry, and in the server-side publish record — and must never be reused or reassigned.

The ARK envelope itself (NAAN, dx shoulder, check character, .<FILE>/?<QUERY>/#<CALLOUT> suffixes) is specified in the ROADMAP; the alphabet and blade lengths are diaryx_ark's, which follows prov's minter.

Import contract

Every importer emits document records. Import is a first-class workspace crate (crates/diaryx_import), not a plugin — the Extism stack that once held it was removed in archive/tauri-web-1.6.1, and the crate archived alongside it targeted the since-deleted diaryx_core. Its parsers are pure (bytes -> records) and port forward; its writer does not, because it emits the Markdown card this spec no longer describes.

The contract:

• ImportedEntry → an entry file (title, date, body, extra frontmatter), for born-digital text with no artifact
• ImportedAttachment → an artifact written through Workspace::import_attachment, which places the payload and lets prov mint the sidecar; the importer then stamps provenance onto that sidecar
• Importers that recover text from an artifact (OCR, speech-to-text) emit a transcription file with transcription_status: draft, linked from the sidecar's transcription:
• Importers populate provenance keys when the source format provides them (email headers → people, EXIF origin date → date_of_document, etc.), and write date_of_document: unknown when the source offers no date for the original — never today's date, and never a date inferred from created

Nothing an importer writes may claim more than it knows. Deterministic decodes of born-digital originals (e.g. the text part of an email) may enter as transcription_status: verified; OCR and speech-to-text must enter as draft, per the lifecycle above.

Future directions (deliberately deferred)

• Entity pages: people/places should eventually be links to entity pages rather than bare strings. Bare strings are forward-compatible with this (a string can be upgraded to a link), so start simple.
• Transcription layer typing (verbatim vs. normalized) — needs real usage first.
• Page-level mapping between multi-page artifacts and transcript sections.